Creating an Annotated Set of Medical Reports to Evaluate Information Retrieval Techniques

During the evaluation of Information Retrieval Algorithms, measurements are often made against a Gold Standard. Since the construction of such a Gold Standard requires considerable resources, a high validity and a high degree of reusability would be an advantage. From a scientific point of view, a high comparability between different Gold Standards would be advantageous, in order to enable the comparison of the results of different measurements. Unfortunately, the validity of such a standard can be negatively affected by many different factors and the reusability and comparability is often limited. In this article, we enumerate some problems, which can negatively affect the result. We show, by the creation of an annotated set of pathology reports, in which form these problems can emerge and how we have tried to minimize their influence.